Papers with composing evaluation guidelines
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. |
| Approach: | They propose to collect annotated human evaluation guidelines and a method for detecting guideline vulnerabilities using Large Language Models. |
| Outcome: | The proposed dataset includes eight vulnerabilities and a method for detecting guideline vulnerabilities. |